Skip to content

Recover PGlite data dirs with corrupt WAL#994

Open
yestheboxer wants to merge 1 commit into
electric-sql:mainfrom
yestheboxer:codex/recover-corrupt-wal
Open

Recover PGlite data dirs with corrupt WAL#994
yestheboxer wants to merge 1 commit into
electric-sql:mainfrom
yestheboxer:codex/recover-corrupt-wal

Conversation

@yestheboxer
Copy link
Copy Markdown

@yestheboxer yestheboxer commented May 19, 2026

This adds a recovery path for NodeFS data directories that fail during startup because Postgres reports corrupt WAL or checkpoint state. Instead of surfacing only an Emscripten abort, PGlite now captures the startup logs, resets WAL in the existing data directory, and retries startup once.

The reset code is intentionally narrow. It only runs for PGlite's PostgreSQL 17 layout, validates pg_control and WAL sizes, removes stale postmaster.pid and old WAL files, writes a replacement shutdown checkpoint record, and preserves the data files. Users can opt out with dataDirRepair: 'none'. The instance exposes repairedDataDir when recovery happened.

The regression test creates a database, inserts a row, truncates WAL, reopens the same data directory, and verifies the original row is still readable after recovery. It also checks that the opt out path still fails with a startup error.

Validation run:

pnpm --filter @electric-sql/pglite build
pnpm --filter @electric-sql/pglite typecheck
pnpm vitest tests/startup-wal-repair.test.ts
/tmp/bun-versions/stable-1.3.14/bun-darwin-aarch64/bun test --timeout 15000 tests/startup-wal-repair.test.ts
/tmp/bun-versions/canary/bun-darwin-aarch64/bun test --timeout 15000 tests/startup-wal-repair.test.ts
pnpm exec prettier --check .changeset/repair-pglite-corrupt-wal.md packages/pglite/src/fs/base.ts packages/pglite/src/fs/nodefs.ts packages/pglite/src/fs/pgResetWal.ts packages/pglite/src/interface.ts packages/pglite/src/pglite.ts packages/pglite/tests/startup-wal-repair.test.ts
git diff --check

@yestheboxer yestheboxer mentioned this pull request May 19, 2026
Open
@yestheboxer yestheboxer marked this pull request as ready for review May 19, 2026 22:15
@rvdlaar
Copy link
Copy Markdown

rvdlaar commented May 23, 2026

Chiming in with an independent reproduction and a deployment data point on this branch.

Setup

PGLite engine inside gbrain (a Postgres-native personal knowledge brain), used as the storage for a long-running HTTP MCP server under launchd on macOS 26.x (Darwin 25.5.0) arm64 / Bun 1.3.11. ~520-page corpus, 80MB on disk. Single-writer model: the daemon holds the exclusive lock for hours; CLI tools occasionally grab it briefly for inserts.

Bug profile, without #994

Five brain.pglite.broken-* snapshots over six days:

Date Trigger
2026-05-17 Background activity (no manual restart)
2026-05-18 Background activity
2026-05-19 Background activity
2026-05-22 (×2) Manual MCP restart cycles during a debug session

Pre-session organic rate: ~3 wedges in 5 days under normal operation, matching the launchd-managed MCP's restart cadence. During a 3-hour debug session with ~6 manual launchctl bootout/bootstrap cycles on a heavily-written brain, additional wedges fired in tight correlation with the restart cycles.

Failure signature is the one this PR's body describes: PGLite failed to initialize its WASM runtime. Aborted(). Build with -sASSERTIONS for more info. on next open. Per-incident state matched what's reported on #223 by intenseandnoble, William, and yestheboxer:

  • postmaster.pid had survived prior shutdown(s)
  • pg_wal/ non-empty
  • No concurrent writer at corruption time

Converged on the same diagnosis: launchd SIGTERM → PGLite-shutdown-checkpoint race leaves WAL torn at shutdown, replay can't reconcile on next open, Emscripten Aborted() is the terminal symptom rather than the cause. macOS/WASM is innocent (rebuild-on-same-OS recovers, as #223 commenters established).

Branch deployment

Built head b01fcda of codex/recover-corrupt-wal via pnpm install && tsup && bundle-wasm and overlaid the resulting .js/.cjs on top of npm-released @electric-sql/pglite@0.4.5 (WASM/data/tarballs preserved). gbrain runs unmodified against the patched dist: startup, list_pages, hybrid search, embed pipeline, and a brainstorm end-to-end run all work without complaint.

The recovery path itself hasn't fired in anger yet — no wedge has occurred in the few hours since the overlay landed. Plan to follow up with an in-anger observation when the next wedge happens, specifically whether repairedDataDir lands on the instance and whether indexed-page survival matches expectation.

What I can offer back

  1. Recovery-path-in-anger observation. Will report back the first time the patched path fires on a real wedge, including repairedDataDir value, page-count delta vs. pre-wedge, and behavior under follow-up writes.
  2. Real-corpus fixture. If there's a torn-WAL scenario you'd want exercised beyond the 1-row insert in tests/startup-wal-repair.test.ts, the 80MB brain that's seen this failure mode in the wild could feed targeted-corruption test fixtures.
  3. Rebase. PR is marked mergeStateStatus: DIRTY per the API. Happy to push a rebased branch if useful; just let me know.

Thanks for the work — the recovery scope (PG17-only layout, opt-out via dataDirRepair: 'none', preserves data files, surfaces repairedDataDir) reads carefully bounded, which I appreciate given the asymmetric blast radius of a misfire.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@yestheboxer @rvdlaar